A storage controller is a physical processing device that is used to store and retrieve data on behalf of one or more hosts. A network storage controller can be configured (e.g., by “hardwiring”, software, firmware, or any combination thereof) to operate as a storage server that serves one or more clients on a network, to store and manage data in a set of mass storage devices, such as magnetic or optical storage-based disks, tapes, or flash memory. Some storage servers are designed to service file-level requests from hosts, as is commonly the case with file servers used in a network attached storage (NAS) environment. Other storage servers are designed to service block-level requests from hosts, as with storage servers used in a storage area network (SAN) environment. Still other storage servers are capable of servicing both file-level requests and block-level requests, as is the case with certain storage servers made by NetApp®, Inc. of Sunnyvale, Calif., employing the Data ONTAP® storage operating system.
As storage systems become larger, in order to accommodate the need for more capacity, several problems arise. Particularly, the efficient use of storage space becomes more difficult. An example of inefficient use of storage space is duplication of data. Deduplication operations eliminate the redundant data objects and instead reference just the original object. These operations provide a benefit in storage space efficiency. The result can be reduced cost in storage acquisition or longer intervals between storage capacity upgrades and more efficient management of data stored on the volumes. Deduplication and other storage efficiency functions can be performed by a storage controller.
A network storage system can have a simple architecture; for example, an individual storage server can provide one or more clients on a network with access to data stored in a mass storage subsystem. Recently, however, with storage capacity demands increasing rapidly in almost every business sector, there has been a trend towards the use of clustered network storage systems, to improve scalability.
In a clustered storage system, two or more storage server “nodes” are connected in a distributed architecture. The nodes are generally implemented by two or more storage controllers. Each storage server “node” is in fact a storage server, although it is implemented with a distributed architecture. For example, a storage server node can be designed to include a network module (“N-module”) to provide network connectivity and a separate data module (e.g., “D-module”) to provide data storage/access functionality, where the N-module and D-module communicate with each other over some type of physical interconnect. Two or more such storage server nodes are typically connected to form a storage “cluster”, where each of the N-modules in the cluster can communicate with each of the D-modules in the cluster.
A clustered architecture allows convenient scaling through the addition of more N-modules and D-modules, all capable of communicating with each other. Further, a storage cluster may present a single system image of stored data to clients and administrators, such that the actual location of data can be made transparent to clients and administrators. An example of a storage controller that is designed for use in a clustered system such as this is a NetApp FAS-series Filer employing NetApp's Data ONTAP® GX storage operating system.
Efficient use of storage space can also be a concern in a clustered storage system. The problem can even be magnified due to the distributed architecture of a clustered storage system. Because of the distributed architecture, the storage a client accesses may not all be controlled by the same D-module. Further, a single D-module may control storage accessed by multiple clients and managed by administrators in multiple locations. These multiple administrators may simultaneously request storage efficiency operations to be performed by the D-module. Storage efficiency operations are operations performed by the D-module to improve the way storage space is used. Deduplication and compression operations are examples of storage efficiency operations. However, each D-module has a limited amount of processing resources to perform the requested storage efficiency operations. If too many storage efficiency operation requests are made simultaneously, the processing resources on the D-module become overloaded and system performance for all users will be degraded.
The problem then becomes how to provide remote administrators the ability to schedule the limited storage efficiency resources, while avoiding resource over-commitment caused by too many concurrent requests for the processors on a given D-module.